{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9d630b4f",
   "metadata": {},
   "source": [
    "# Lab: Chicago taxifare tip prediction with AutoML Tables on Vertex Pipelines using Kubeflow Pipelines SDK"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a978d653",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "1. Perform exploratory data analysis (EDA) on tabular data using BigQuery.\n",
    "2. Create a BigQuery dataset for a ML classification task.\n",
    "3. Define an AutoML tables pipeline using the Kubeflow Pipelines (KFP) SDK for model training, evaluation, and conditional deployment.\n",
    "4. Create a custom model evaluation component using the KFP SDK.\n",
    "5. Incorporate pre-built KFP components into your pipeline from `google_cloud_components`.\n",
    "6. Query your model for online predictions and explanations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89e15e7f",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb9f1f2",
   "metadata": {},
   "source": [
    "### Define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0c15b0e0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PATH=/usr/local/cuda/bin:/opt/conda/bin:/opt/conda/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/jupyter/.local/bin\n"
     ]
    }
   ],
   "source": [
    "# Add installed depedencies to Python PATH variable.\n",
    "PATH=%env PATH\n",
    "%env PATH={PATH}:/home/jupyter/.local/bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0cc9968e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dougkelly-vertex-demos\n",
      "us-central1\n"
     ]
    }
   ],
   "source": [
    "PROJECT_ID = !(gcloud config get-value core/project)\n",
    "PROJECT_ID = PROJECT_ID[0]\n",
    "REGION = 'us-central1'\n",
    "\n",
    "BQ_DATASET_NAME = 'chicago_taxi'\n",
    "BQ_TABLE_NAME = 'chicago_taxi_tips_raw'\n",
    "BQ_LOCATION = 'US'\n",
    "\n",
    "!echo {PROJECT_ID}\n",
    "!echo {REGION}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83db0b33",
   "metadata": {},
   "source": [
    "### Create Cloud Storage bucket for storing Vertex Pipeline artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "472c22fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://dougkelly-vertex-demos-bucket\n"
     ]
    }
   ],
   "source": [
    "BUCKET_NAME = f\"gs://{PROJECT_ID}-bucket\"\n",
    "print(BUCKET_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6cd6c5d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                 gs://dougkelly-vertex-demos-bucket/pipeline_root/\n"
     ]
    }
   ],
   "source": [
"!gcloud storage ls --all-versions --long $BUCKET_NAME"   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "50f69311",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'gs://dougkelly-vertex-demos-bucket/pipeline_root/dougkelly'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "USER = \"dougkelly\"  # <---CHANGE THIS\n",
    "PIPELINE_ROOT = \"{}/pipeline_root/{}\".format(BUCKET_NAME, USER)\n",
    "\n",
    "PIPELINE_ROOT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc486365",
   "metadata": {},
   "source": [
    "### Create BigQuery dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "26ac730f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BigQuery error in mk operation: Dataset 'dougkelly-vertex-demos:chicago_taxi'\n",
      "already exists.\n"
     ]
    }
   ],
   "source": [
    "!bq --location=US mk -d \\\n",
    "$PROJECT_ID:$BQ_DATASET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b002020",
   "metadata": {},
   "source": [
    "## Exploratory Data Analysis in BigQuery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "56285d7b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Query complete after 0.01s: 100%|██████████| 3/3 [00:00<00:00, 1198.94query/s]                        \n",
      "Downloading: 100%|██████████| 7/7 [00:01<00:00,  6.01rows/s]\n"
     ]
    }
   ],
   "source": [
    "%%bigquery data\n",
    "\n",
    "SELECT \n",
    "    CAST(EXTRACT(DAYOFWEEK FROM trip_start_timestamp) AS string) AS trip_dayofweek, \n",
    "    FORMAT_DATE('%A',cast(trip_start_timestamp as date)) AS trip_dayname,\n",
    "    COUNT(*) as trip_count,\n",
    "FROM `bigquery-public-data.chicago_taxi_trips.taxi_trips`\n",
    "WHERE\n",
    "    EXTRACT(YEAR FROM trip_start_timestamp) = 2015 \n",
    "GROUP BY\n",
    "    trip_dayofweek,\n",
    "    trip_dayname\n",
    "ORDER BY\n",
    "    trip_dayofweek\n",
    ";"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "be809e85",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAFCCAYAAADYJ5e4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfTElEQVR4nO3de7zVdZ3v8ddbQEFBSdxaRgZamndUxAsOoZZZdjG1acwszfR4tPtUwzlnZkQrj2fEOl2n4Zh5CU0NdcxGy/GIhgqEiCBe8gIWY42IoWhiKp/54/tbuNjsy1qw1/p9f3u/n4/HeqzL77f3/rD5rff+re/ve1FEYGZm+dqs7ALMzKxnDmozs8w5qM3MMuegNjPLnIPazCxzDmozs8y1LKglXSLpaUkPNLj/X0t6UNISSVe2qi4zs6pRq/pRS5oEvABcHhF79bLv24FrgCMi4k+Sto+Ip1tSmJlZxbTsjDoi7gSerX9N0i6SbpF0r6RfS3pHsel04PsR8afiax3SZmaFdrdRTwc+GxEHAF8GflC8viuwq6S7JM2RdHSb6zIzy9bgdv0gScOBQ4FrJdVe3qKujrcDk4HRwK8l7RURq9pVn5lZrtoW1KSz91URMa6LbcuBORHxCrBU0iOk4P5NG+szM8tS25o+IuJ5Ugh/BEDJvsXmG4DDi9e3IzWFPNGu2szMctbK7nlXAfcAu0laLuk04CTgNEn3A0uADxW7/xJYKelB4HbgKxGxslW1mZlVScu655mZWd/wyEQzs8w5qM3MMteSXh/bbbddjBkzphXf2sysX7r33nufiYiOrra1JKjHjBnD/PnzW/Gtzcz6JUlPdrfNTR9mZplzUJuZZc5BbWaWubYNIX/llVdYvnw5a9asadeP7PeGDh3K6NGjGTJkSNmlmFkLtS2oly9fzogRIxgzZgx1kzLZRooIVq5cyfLlyxk7dmzZ5ZhZC7Wt6WPNmjWMGjXKId1HJDFq1Ch/QjEbANraRu2Q7lv+fZoNDL6YaGaWuXbOR72eMVN+0affb9kFx/S4fdWqVVx55ZWcddZZXW4/9NBDufvuu/u0pr5w6aWXctRRR7HjjjuWXYrZBvr6fdxZb+/rgWLAnFGvWrWKH/zgBxu8/tprrwFkGdKQgvqpp54quwwzK9GACeopU6bw+OOPM27cOA488EAOP/xwPvaxj7H33nsDMHz4cABmzZrFpEmT+PCHP8wee+zBmWeeydq1a7v9vrfccgv7778/++67L0ceeSQAzz77LMceeyz77LMPBx98MIsWLQJg6tSpTJs2bd3X7rXXXixbtoxly5ax++67c/rpp7Pnnnty1FFH8dJLL/Gzn/2M+fPnc9JJJzFu3DheeumlVv16zCxjAyaoL7jgAnbZZRcWLlzIhRdeyLx58/jGN77Bgw8+uMG+8+bN46KLLmLx4sU8/vjjXHfddV1+zxUrVnD66aczc+ZM7r//fq699loAzjnnHPbbbz8WLVrE+eefzyc+8Yle63v00Uc5++yzWbJkCSNHjmTmzJmccMIJjB8/nhkzZrBw4UKGDRu2ab8EM6ukARPUnU2YMKHb/scTJkxg5513ZtCgQZx44onMnj27y/3mzJnDpEmT1n2fbbfdFoDZs2dz8sknA3DEEUewcuVKnnvuuR7rGTt2LOPGjQPggAMOYNmyZRvxrzKz/mjABvVWW23V7bbO3d666wYXEV1u62rVHEkMHjx4vWaU+j7QW2yxxbrHgwYN4tVXX+2+eDMbUAZMUI8YMYLVq1c3tO+8efNYunQpa9eu5eqrr+awww7rcr9DDjmEO+64g6VLlwKpbRpg0qRJzJgxA0ht3ttttx1bb701Y8aMYcGCBQAsWLBg3df1Vd1m1j+V1j2v3d1uRo0axcSJE9lrr70YNmwYO+ywQ7f7HnLIIUyZMoXFixevu7DYlY6ODqZPn85xxx3H2rVr2X777bn11luZOnUqp556Kvvssw9bbrkll112GQDHH388l19++boLmrvuumuvdZ9yyimceeaZDBs2jHvuucft1GYDUEOL20paBqwGXgNejYjxPe0/fvz46LxwwEMPPcTuu+++8ZW2yaxZs5g2bRo33XRT2aU0pCq/V+uf3I+670i6t7tsbeaM+vCIeKaPajIzswaV1vSRq8mTJzN58uQNXj/ooIN4+eWX13vtiiuuWNcP28ysVRoN6gB+JSmAf4mI6Z13kHQGcAbATjvt1PU36aaXRBXMnTu37BI20EizlZl1r5VNN33ZbNNor4+JEbE/8F7gbEmTOu8QEdMjYnxEjO/o2HAh3aFDh7Jy5UqHSx+pzUc9dOjQsksxsxZr6Iw6Ip4q7p+WdD0wAbizmR80evRoli9fzooVK5qv0rpUW+HFzPq3XoNa0lbAZhGxunh8FHBesz9oyJAhXonErBP3mrBGNHJGvQNwfdG2PBi4MiJuaWlVZma2Tq9BHRFPAPu2oRYzM+vCgBlCbmZWVQ5qM7PMOajNzDLnoDYzy5yD2swscw5qM7PMOajNzDLnoDYzy5yD2swscw5qM7PMOajNzDLnoDYzy5yX4rJK8zShNhD4jNrMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzLl73gDn7m1m+fMZtZlZ5hzUZmaZc1CbmWXOQW1mljkHtZlZ5hzUZmaZc1CbmWXOQW1mlrksBry0ctCFB1yYWdVlEdRV5pF9ZtZqDTd9SBok6T5JN7WyIDMzW18zbdSfBx5qVSFmZta1hoJa0mjgGODi1pZjZmadNXpG/X+BrwJru9tB0hmS5kuav2LFir6ozczMaCCoJb0feDoi7u1pv4iYHhHjI2J8R0dHnxVoZjbQNXJGPRH4oKRlwE+BIyT9pKVVmZnZOr0GdUT8j4gYHRFjgL8B/n9EfLzllZmZGeCRiWZm2WtqwEtEzAJmtaQSMzPrks+ozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8tcr0EtaaikeZLul7RE0rntKMzMzJLBDezzMnBERLwgaQgwW9LNETGnxbWZmRkNBHVEBPBC8XRIcYtWFmVmZq9rqI1a0iBJC4GngVsjYm5LqzIzs3UaCuqIeC0ixgGjgQmS9uq8j6QzJM2XNH/FihV9XKaZ2cDVVK+PiFgFzAKO7mLb9IgYHxHjOzo6+qY6MzNrqNdHh6SRxeNhwLuAh1tcl5mZFRrp9fEm4DJJg0jBfk1E3NTasszMrKaRXh+LgP3aUIuZmXXBIxPNzDLnoDYzy5yD2swscw5qM7PMOajNzDLnoDYzy5yD2swscw5qM7PMOajNzDLnoDYzy5yD2swscw5qM7PMOajNzDLnoDYzy5yD2swscw5qM7PMOajNzDLnoDYzy5yD2swscw5qM7PMOajNzDLnoDYzy5yD2swscw5qM7PMOajNzDLnoDYzy5yD2swscw5qM7PMOajNzDLnoDYzy1yvQS3pLZJul/SQpCWSPt+OwszMLBncwD6vAn8bEQskjQDulXRrRDzY4trMzIwGzqgj4g8RsaB4vBp4CHhzqwszM7OkqTZqSWOA/YC5LanGzMw20HBQSxoOzAS+EBHPd7H9DEnzJc1fsWJFX9ZoZjagNRTUkoaQQnpGRFzX1T4RMT0ixkfE+I6Ojr6s0cxsQGuk14eAHwEPRcQ3W1+SmZnVa+SMeiJwMnCEpIXF7X0trsvMzAq9ds+LiNmA2lCLmZl1wSMTzcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8ucg9rMLHMOajOzzDmozcwy56A2M8tcr0Et6RJJT0t6oB0FmZnZ+ho5o74UOLrFdZiZWTd6DeqIuBN4tg21mJlZF9xGbWaWuT4LaklnSJovaf6KFSv66tuamQ14fRbUETE9IsZHxPiOjo6++rZmZgOemz7MzDLXSPe8q4B7gN0kLZd0WuvLMjOzmsG97RARJ7ajEDMz65qbPszMMuegNjPLnIPazCxzDmozs8w5qM3MMuegNjPLnIPazCxzDmozs8w5qM3MMuegNjPLnIPazCxzDmozs8w5qM3MMuegNjPLnIPazCxzDmozs8w5qM3MMuegNjPLnIPazCxzDmozs8w5qM3MMuegNjPLnIPazCxzDmozs8w5qM3MMuegNjPLnIPazCxzDmozs8w5qM3MMuegNjPLnIPazCxzDQW1pKMlPSLpMUlTWl2UmZm9rtegljQI+D7wXmAP4ERJe7S6MDMzSxo5o54APBYRT0TEX4CfAh9qbVlmZlajiOh5B+kE4OiI+HTx/GTgoIj4TKf9zgDOKJ7uBjzS9+UCsB3wTIu+dzu4/nK5/nJVuf5W1/7WiOjoasPgBr5YXby2QbpHxHRgepOFNU3S/IgY3+qf0yquv1yuv1xVrr/M2htp+lgOvKXu+WjgqdaUY2ZmnTUS1L8B3i5prKTNgb8BbmxtWWZmVtNr00dEvCrpM8AvgUHAJRGxpOWVda/lzSst5vrL5frLVeX6S6u914uJZmZWLo9MNDPLnIPazCxzDmrrkaS9yq5hIJO0bdk1WPkqEdTFMPbKqnj9P5Q0T9JZkkaWXcwANFfStZLeJ6mrMQ3Zq/jxn4VKBDXwmKQLKzzHSGXrj4jDgJNIfennS7pS0rtLLqthkj4j6Q1l17EJdiX1NjiZdBydL2nXkmtqVmWPf0nTJO1Zdh1VCep9gN8CF0uaI+kMSVuXXVQTKl1/RDwK/D3wd8A7ge9IeljSceVW1pA3Ar+RdE0xC2SlzkojuTUiTgQ+DXwSmCfpDkmHlFxeo6p8/D8MTJc0V9KZkrYppYqIqNQNmAT8B/AicBnwtrJr6s/1k95k3yK90b4P7F+8viPwZNn1NfhvEPAe0oRijwHnA7uUXVeDtY8CPg/MB34BHEca/zAeWFp2fRvx76nU8V9X927ABcCTwJXA4e38+ZU4o5Y0SNIHJV0PfBu4CNgZ+Dnwb6UW14CK1/89YAGwb0ScHRELACLiKdJZdvYivdP+WNxeBd4A/EzSP5VaWGPuAbYGjo2IYyLiuoh4NSLmAz8subaGVPz4r7Wxv6O4PQPcD3xJ0k/bVkPx1yJrkp4Abgd+FBF3d9r2nYj4XDmVNabq9VeZpM+RmgueAS4GboiIVyRtBjwaEbuUWmAvJCmq8CbtQZWPf0nfBD4I3Eaqf17dtkciYre21FGFY0DS8Ih4oew6NlaV65f0duB/kxaNGFp7PSJ2Lq2oJkg6j/QGe7KLbbtHxEMllNUwSR3AV4E9Wf/3f0RpRTWp4sf/p4CfRsSfu9i2TUQ815Y6KhLUQ4HT2PBg/VRpRTWhyvVLmg2cQ2qn/gBwKum4OafUwpokaXvW/93/rsRyGibpV8DVwJeBM0mfDlZExN+VWlgTqnz8AxS9ht7O+rXf2c4aKtFGDVxBunr/HuAO0lSrq0utqDlVrn9YRNxGCucnI2IqUKWzuQ9IehRYSvrdLwNuLrWo5oyKiB8Br0TEHUW4HVx2UU2q7PEv6dPAnaRJ6c4t7qe2u46qBPXbIuIfgBcj4jLgGGDvkmtqRpXrX1Nrzy36JH8Y2L7soprwdVKw/TYixgJHAneVW1JTXinu/yDpGEn7kYKuSqp8/H8eOJDUw+lwYD9gRbuLqEpQ1w7WVcWQ5m2AMeWV07Qq1/8FYEvgc8ABpIEXnyyzoCa9EhErgc0kbRYRtwPjSq6pGV8v+u7+Lan542Lgi+WW1LQqH/9rImINgKQtIuJhUle9tmpkKa4cTC/aif6BtGjBcOAfyy2pKZWtPyJ+Uzx8gdQ+XTWrJA0nfXydIelpUhe9SoiIm4qHzwGHl1nLJqjs8Q8sL6ZOuAG4VdKfKGGFq0pcTLT2k/RzulgbsyYiPtjGcjaapK2ANaRBLyeRzuZmFGfZ2ZL0XXr+/Wfbpa2/kvRO0vFzS0T8pZ0/O+szaklf6ml7RHyzXbVsjIrXP624P450IegnxfMTSRfkKiEiXqx7ellphTRvfnE/kdQ18uri+UeAe0upqElVPv67mbVwcXE/HHi2jeXkHdTAiOJ+N1KDfm2txg+QPsrmrrL1R8QdAJK+FhGT6jb9XFLWtQNIWk3PZ6RZzzVRXHRD0imk4cqvFM9/CPyqxNKaUdnjn/THMEifxHYC/lQ8Hgn8DhjbzmKyDuqIOBfW9SXdPyJWF8+nAteWWFpDql5/oUPSzhHxBICksUBHyTX1KiJGwLoBL38kdRGrNX+M6OFLc7Mjqd7aGdzw4rXsVfn4L3oI1f4w3hgR/1Y8fy/wrnbXk3VQ19kJqG8T+gvVuWoM1a7/i8CsYhgwpLr/W3nlNO09EXFQ3fN/ljQXqMI8H5AmArpP0u3F83dSQj/eTVTl4//AiDiz9iQibpb0tXYXUZWgvoI0teP1pI8jHwYuL7ekplS2/oi4pRhG/o7ipYcj4uUya2rSa5JOIs2cF6Q29tfKLalxEfFjSTcDtT82UyLij2XWtBEqe/wDz0j6e9I1mgA+DrT9QnRlen1IOgA4rHh6Z0TcV2Y9zapq/ZI+QrrKvbo4YPcHvl6bRS93ksaQZmybSHqj3QV8ISKWlVhWwyRNBBZGxIuSPk76/X+7q7lLciZpf+CviqdVOv63JU2hMIl0/NwJnBcRbb2YWKWgHgTsQN2ngKrM1wDVrV/SoojYR9JhpMmZpgH/s1NzgrWIpEXAvqR5wS8HLgGOi4h3llpYAyRtHRHPd9ODgnaHXbOK9+xlEfHxsmupxMhESZ8F/hO4FbiJNIH6TT1+UUYqXn+tmeAY4J8j4l+BzUuspymS/knS1pKGSLpN0jPFmWlVvFpMc/oh4DsR8W2qczH0yuL+XlJ3w9qt9jxrEfEa6WJ66cd7Jc6oJT0GHJT7IIXuVLl+STeRVuR4F2kI+UvAvIjYt9TCGiRpYUSMK+YoOZZ0cfT2CtV/B3ALaVToJNI8EwsjohJzZUgS8JYqfHrsiqR/ITU33UhalQZofx/wSpxRA78nDaGtqirX/9ekGcOOjohVwLbAV0qtqDlDivv3AVfl/nG7Cx8FXgZOKy4ivhm4sNySGld8Gri+7Do2wVOkT7+bkT7J1G5tVZVeH0+Quoj9gnTQAnmPbOqksvVHxJ+L+TEOAx4lzZPxaLlVNeXnkh4mfRI4q5iIf03JNTWkaCP9SUSs67dbnJlWpcdEzRxJB9bNG1MZtb7gZatKUP+uuG1OhdpH61S2fknnkBZS3Q34MekM9SekXhTZi4gpkv4P8HxEvCbpz6T23uzV6m3nSiItcjhwpqRlpOYDkU629ym1qgYU/dc3aB9u9wo7lWij7i8kjSAdoJVZlkjSQtIcvAsiYr/itUVVeJMBSNoS+BKwU0ScUfQJ361uVrqsSbqGNJ/2razfRpr9pEySdoqI30l6a1fbq9DFsOhWWzMUOJ50gfer7ayjEmfUufxV21jFHLxXkNp3kfQM8ImIWFJqYY35S0SEpIB1s9FVyY9JvQwOLZ4vJw1frkRQk3oI/aLsIjbSDaSh409KmhkRx5ddULMiovMEWHcVF3jbqhJBTZowvWbdX7WSatkY04EvFZPWI2ky8P94PTxydk1x5XukpNOBT5Fqr4pdIuKjkk4EiIiXip4IlVCbnKmi6n/PlVgMubNOfcA3I/V8emO766hEUOfyV20TbFULaYCImFWVM9OImCbp3cDzpHbqf4yIW0suqxl/kTSM4hOZpF2ou6CbO0lL6frTZBWCL7p5XCX1s+i9Slp787R2F1GJoO7ir9p4SvirtgmekPQPpOYPSPMFLC2xnqYUwVylcK53Dqkf8lskzSBdBD2l1IqaM77u8VDSfNRdjvTL0L6SnieF3LDiMbx+MTHrqWYLu9eW4qqRtEW7i6jExcROZxWvkiauPy8iZpdWVBOKZYjOJXVxE2m+gKkR8adSC2tAp3mdNyf1+nixIm8yACSNIl2QEzAnIp4puaRNIml2RBzW+562qSQtiIj9e3ut1bI+o5Z0IPD7urlhP0lqn14GPFhiaU0pAjn7q/Rdqc3rXCPpWGBCOdVstKGkid8HA3tIIiJyn7geWDeZUU3t02RVhpBXlqQ3kgYXDVNa+b3W3r41abHn9taT8xm1pAXAuyLiWUmTSFNVfpa0ivTuEXFCmfX1RtKNPW2PjNcdlDQ4Irq8YCtpTkQc3O6aNkbRh/qjwBJgbfFy5Py7r1c3DzW8/mlyWkQ8Uk5FA0NxUngK6Q9j/bwkq4FLI+K6ttaTeVDfX5uTQdL3gRURMbV4vjAixpVYXq8krSANH78KmMv6V8HXLXeVo9rHO0nH1b1cO6N7Z0QcUlJpTZH0CLBPxebQtkxIOj4iZpZdR9ZNH8CgujO7I4Ez6rblXjukC57vJk1W/zFSf9irKtJ/uuYDbHh9oBJno4UnSO3qlQzq4sLV8aQVUeqnyD2vrJoGkoiYKekYYE9SE1rt9bb+/nMPu6uAO4oBIi8BvwaQ9DYqMMlRMU3iLcAtxRvuRNKcH+dFxHfLra5X2yutIv1Ap9cDOBnIfp6Swp+BhZJuY/15VqpyzeBfScf6vVT0j02VKa2ZuCVpGPzFwAnAvHbXkXVQR8Q3ijfYm4BfxevtNJuR2qqzVwT0MaSQHgN8B2hr+9ZGGkRaSLUyg0O6cSOvr35dRaMj4uiyixjADi0WzlgUEedKuogS3r9ZBzVARMzp4rXfllFLsyRdBuwF3AycGxGdz05z9of+8PG64iP7AO6WtHdELC67kAHqpeL+z5J2JK0GP7bdRWQf1BV3MmkinV2Bz9WNXK5Ch/9Kn0lLWkwPo+Fyn1RK0gOkXiqDgVOVVoF/mQrNPNdP3CRpJGnV+toI6YvbXYSDuoUioioLM3TlyLIL2ETvL+7PLu5ro0JPIrVb5+7NpG6oVoK6MRxfK54PBxYDDwPfans9OXfPM9tUku6KiIm9vZabMka/2etyG8PhM2rr77aSdFhtugFJhwJVmBCr1uumS1VYHajiBtUt2/ZRYHrRn3pmMUd7Wzmorb87DbhE0jakNuvnSFO15q6/9LqpqqzGcDiorV8rpsjdV9LWpKa+7PvfF/pFr5sKy2oMh9uorV+TtANwPrBjRLxX0h7AIRHxo5JL65Gk+2pLn1k5JB3M62M4Xixe2xUYHhEL2lqLg9r6M0k3k5bj+l8Rsa+kwcB9EbF3yaX1SNK2dW2kNsBVufuYWSO2i4hrKGbOK9ocXyu3pN45pK2eg9r6uxeLhQNqS3EdTAXmiTGr56YP65ckfQG4i9Rr4pukofxLgA7gIxFxf3nVmTXHQW39kqRppFXe30EaTfYfwCzg6qovxWUDj4Pa+jVJm5MWOzgUOKS4rYqIPUotzKwJ7kdt/d0w0jp32xS3p0hzNphVhs+orV+SNJ20Ksdq0jJoc0grkGe/8rtZZ+71Yf3VTsAWwB9J7dPLgVVlFmS2sXxGbf2W0gTge5Lapw8l9fx4FrgnIs4pszazZjiord+TNBqYSArr9wOjImJkqUWZNcFBbf2SpM+Rgnki8AqpT/U9xf3iiFhbYnlmTXGvD+uvxgA/A74YEX8ouRazTeIzajOzzLnXh5lZ5hzUZmaZc1BbW0gaKemsHrbf3Qc/4xRJ39vU72OWGwe1tctIYIOgljQIICIObXdBZlXhoLZ2uQDYRdJCSb+RdLukKynm3ZD0QnE/WdKdkq6X9KCkH0rq9jiVdKqk30q6g9QVr/b6ByTNlXSfpH+XtIOkzSQ9Kqmj2GczSY9J2k7SpZK+I+luSU9IOqHYZ7ik2yQtkLRY0oeK18dIeljSxZIekDRD0rsk3VX8jAnFfltJuqT4N99X+3qzpkSEb761/EbqLvdA8Xgy8CIwtm77C3Xb1gA7k1bivhU4oZvv+Sbgd6Q5pjcn9ZH+XrHtDbzeq+nTwEXF43OALxSPjwJmFo8vBa4lnbzsATxWvD4Y2Lp4vB3wGGmO6zHAq8DexdfcC1xSbPsQcEPxNecDHy8ejwR+C2xV9v+Hb9W6+YzayjIvIpb2sO2JiHiNtBr0Yd3sdxAwKyJWRMRfgKvrto0GfilpMfAV0lBySGH6ieLxp0jrKdbcEBFrI+JBYIfiNQHnS1oE/Dvw5rptSyOiNnhmCXBbRATpU8KYYp+jgCmSFpLmwx5KmofErGEe8GJlebGHbZ079/fU2b+7bd8FvhkRN0qaDEwFiIjfS/pPSUeQgv6kuq95ue6xivuTSGfsB0TEK5KWkcK28/5r656v5fX3loDjI+KRHv4NZj3yGbW1y2pgRIP7TpA0tmib/igwu5v95gKTJY2SNAT4SN22bUiz5gF8stPXXQz8BLimOGvvyTbA00VIHw68tcF/Q80vgc8WE0Qhab8mv97MQW3tERErgbskPQBc2Mvu95AuPj4ALAWu7+Z7/oF0pnwPqVliQd3mqcC1kn4NdF5660ZgOOs3e3RnBjBe0nzS2fXDDXxNva8BQ4BFxb/9a01+vZmHkFteimaKL0fE+1v4M8YD34qIv2rVzzDrS26jtgFF0hTgv7N+27RZ1nxGbZUgaS5pxZZ6J0eE1z+0fs9BbWaWOV9MNDPLnIPazCxzDmozs8w5qM3MMuegNjPL3H8BlSFe+h476F4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.plot(kind='bar', x='trip_dayname', y='trip_count');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95e58013",
   "metadata": {},
   "source": [
    "### Create BigQuery dataset for ML classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d69c1167",
   "metadata": {},
   "outputs": [],
   "source": [
    "SAMPLE_SIZE = 100000\n",
    "YEAR = 2020"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "e102f92d",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql_script = '''\n",
    "CREATE OR REPLACE TABLE `@PROJECT_ID.@DATASET.@TABLE` \n",
    "AS (\n",
    "    WITH\n",
    "      taxitrips AS (\n",
    "      SELECT\n",
    "        trip_start_timestamp,\n",
    "        trip_seconds,\n",
    "        trip_miles,\n",
    "        payment_type,\n",
    "        pickup_longitude,\n",
    "        pickup_latitude,\n",
    "        dropoff_longitude,\n",
    "        dropoff_latitude,\n",
    "        tips,\n",
    "        fare\n",
    "      FROM\n",
    "        `bigquery-public-data.chicago_taxi_trips.taxi_trips`\n",
    "      WHERE 1=1 \n",
    "      AND pickup_longitude IS NOT NULL\n",
    "      AND pickup_latitude IS NOT NULL\n",
    "      AND dropoff_longitude IS NOT NULL\n",
    "      AND dropoff_latitude IS NOT NULL\n",
    "      AND trip_miles > 0\n",
    "      AND trip_seconds > 0\n",
    "      AND fare > 0\n",
    "      AND EXTRACT(YEAR FROM trip_start_timestamp) = @YEAR\n",
    "    )\n",
    "\n",
    "    SELECT\n",
    "      trip_start_timestamp,\n",
    "      EXTRACT(MONTH from trip_start_timestamp) as trip_month,\n",
    "      EXTRACT(DAY from trip_start_timestamp) as trip_day,\n",
    "      EXTRACT(DAYOFWEEK from trip_start_timestamp) as trip_day_of_week,\n",
    "      EXTRACT(HOUR from trip_start_timestamp) as trip_hour,\n",
    "      trip_seconds,\n",
    "      trip_miles,\n",
    "      payment_type,\n",
    "      ST_AsText(\n",
    "          ST_SnapToGrid(ST_GeogPoint(pickup_longitude, pickup_latitude), 0.1)\n",
    "      ) AS pickup_grid,\n",
    "      ST_AsText(\n",
    "          ST_SnapToGrid(ST_GeogPoint(dropoff_longitude, dropoff_latitude), 0.1)\n",
    "      ) AS dropoff_grid,\n",
    "      ST_Distance(\n",
    "          ST_GeogPoint(pickup_longitude, pickup_latitude), \n",
    "          ST_GeogPoint(dropoff_longitude, dropoff_latitude)\n",
    "      ) AS euclidean,\n",
    "      CONCAT(\n",
    "          ST_AsText(ST_SnapToGrid(ST_GeogPoint(pickup_longitude,\n",
    "              pickup_latitude), 0.1)), \n",
    "          ST_AsText(ST_SnapToGrid(ST_GeogPoint(dropoff_longitude,\n",
    "              dropoff_latitude), 0.1))\n",
    "      ) AS loc_cross,\n",
    "      IF((tips/fare >= 0.2), 1, 0) AS tip_bin,\n",
    "      IF(ABS(MOD(FARM_FINGERPRINT(STRING(trip_start_timestamp)), 10)) < 9, 'UNASSIGNED', 'TEST') AS data_split\n",
    "    FROM\n",
    "      taxitrips\n",
    "    LIMIT @LIMIT\n",
    ")\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "d90ad4c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql_script = sql_script.replace(\n",
    "    '@PROJECT_ID', PROJECT_ID).replace(\n",
    "    '@DATASET', BQ_DATASET_NAME).replace(\n",
    "    '@TABLE', BQ_TABLE_NAME).replace(\n",
    "    '@YEAR', str(YEAR)).replace(\n",
    "    '@LIMIT', str(SAMPLE_SIZE))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "85edcef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(sql_script)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "af5dd341",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.cloud import bigquery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "a5b7bcf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "bq_client = bigquery.Client(project=PROJECT_ID, location=BQ_LOCATION)\n",
    "job = bq_client.query(sql_script)\n",
    "_ = job.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6223359",
   "metadata": {},
   "source": [
    "#### Verify data split proportions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "38a2981f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Query complete after 0.01s: 100%|██████████| 2/2 [00:00<00:00, 305.47query/s]                         \n",
      "Downloading: 100%|██████████| 2/2 [00:01<00:00,  1.55rows/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>data_split</th>\n",
       "      <th>f0_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>UNASSIGNED</td>\n",
       "      <td>90081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TEST</td>\n",
       "      <td>9919</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   data_split    f0_\n",
       "0  UNASSIGNED  90081\n",
       "1        TEST   9919"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery\n",
    "\n",
    "SELECT data_split, COUNT(*)\n",
    "FROM dougkelly-vertex-demos.chicago_taxi.chicago_taxi_tips_raw\n",
    "GROUP BY data_split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68511647",
   "metadata": {},
   "source": [
    "### Create "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfc0f6c2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3ac13bac",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "670d6b40",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import logging\n",
    "from typing import NamedTuple\n",
    "\n",
    "import kfp\n",
    "# from google.cloud import aiplatform\n",
    "from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
    "from kfp.v2 import dsl\n",
    "from kfp.v2.dsl import (ClassificationMetrics, Input, Metrics, Model, Output,\n",
    "                        component)\n",
    "from kfp.v2.google.client import AIPlatformClient"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c57945a1",
   "metadata": {},
   "source": [
    "## Create and run an AutoML Tabular classification pipeline using Kubeflow Pipelines SDK"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14c9a012",
   "metadata": {},
   "source": [
    "### Create a custom KFP evaluation component"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "147e5283",
   "metadata": {},
   "outputs": [],
   "source": [
    "@component(\n",
    "    base_image=\"gcr.io/deeplearning-platform-release/tf2-cpu.2-3:latest\",\n",
    "    output_component_file=\"components/tables_eval_component.yaml\", # Optional: you can use this to load the component later\n",
    "    packages_to_install=[\"google-cloud-aiplatform==1.0.0\"],\n",
    ")\n",
    "def classif_model_eval_metrics(\n",
    "    project: str,\n",
    "    location: str,\n",
    "    api_endpoint: str,\n",
    "    thresholds_dict_str: str,\n",
    "    model: Input[Model],\n",
    "    metrics: Output[Metrics],\n",
    "    metricsc: Output[ClassificationMetrics],\n",
    ") -> NamedTuple(\"Outputs\", [(\"dep_decision\", str)]):  # Return parameter.\n",
    "\n",
    "    \"\"\"This function renders evaluation metrics for an AutoML Tabular classification model.\n",
    "    It retrieves the classification model evaluation generated by the AutoML Tabular training\n",
    "    process, does some parsing, and uses that info to render the ROC curve and confusion matrix\n",
    "    for the model. It also uses given metrics threshold information and compares that to the\n",
    "    evaluation results to determine whether the model is sufficiently accurate to deploy.\n",
    "    \"\"\"\n",
    "    import json\n",
    "    import logging\n",
    "\n",
    "    from google.cloud import aiplatform\n",
    "\n",
    "    # Fetch model eval info\n",
    "    def get_eval_info(client, model_name):\n",
    "        from google.protobuf.json_format import MessageToDict\n",
    "\n",
    "        response = client.list_model_evaluations(parent=model_name)\n",
    "        metrics_list = []\n",
    "        metrics_string_list = []\n",
    "        for evaluation in response:\n",
    "            print(\"model_evaluation\")\n",
    "            print(\" name:\", evaluation.name)\n",
    "            print(\" metrics_schema_uri:\", evaluation.metrics_schema_uri)\n",
    "            metrics = MessageToDict(evaluation._pb.metrics)\n",
    "            for metric in metrics.keys():\n",
    "                logging.info(\"metric: %s, value: %s\", metric, metrics[metric])\n",
    "            metrics_str = json.dumps(metrics)\n",
    "            metrics_list.append(metrics)\n",
    "            metrics_string_list.append(metrics_str)\n",
    "\n",
    "        return (\n",
    "            evaluation.name,\n",
    "            metrics_list,\n",
    "            metrics_string_list,\n",
    "        )\n",
    "\n",
    "    # Use the given metrics threshold(s) to determine whether the model is \n",
    "    # accurate enough to deploy.\n",
    "    def classification_thresholds_check(metrics_dict, thresholds_dict):\n",
    "        for k, v in thresholds_dict.items():\n",
    "            logging.info(\"k {}, v {}\".format(k, v))\n",
    "            if k in [\"auRoc\", \"auPrc\"]:  # higher is better\n",
    "                if metrics_dict[k] < v:  # if under threshold, don't deploy\n",
    "                    logging.info(\n",
    "                        \"{} < {}; returning False\".format(metrics_dict[k], v)\n",
    "                    )\n",
    "                    return False\n",
    "        logging.info(\"threshold checks passed.\")\n",
    "        return True\n",
    "\n",
    "    def log_metrics(metrics_list, metricsc):\n",
    "        test_confusion_matrix = metrics_list[0][\"confusionMatrix\"]\n",
    "        logging.info(\"rows: %s\", test_confusion_matrix[\"rows\"])\n",
    "\n",
    "        # log the ROC curve\n",
    "        fpr = []\n",
    "        tpr = []\n",
    "        thresholds = []\n",
    "        for item in metrics_list[0][\"confidenceMetrics\"]:\n",
    "            fpr.append(item.get(\"falsePositiveRate\", 0.0))\n",
    "            tpr.append(item.get(\"recall\", 0.0))\n",
    "            thresholds.append(item.get(\"confidenceThreshold\", 0.0))\n",
    "        print(f\"fpr: {fpr}\")\n",
    "        print(f\"tpr: {tpr}\")\n",
    "        print(f\"thresholds: {thresholds}\")\n",
    "        metricsc.log_roc_curve(fpr, tpr, thresholds)\n",
    "\n",
    "        # log the confusion matrix\n",
    "        annotations = []\n",
    "        for item in test_confusion_matrix[\"annotationSpecs\"]:\n",
    "            annotations.append(item[\"displayName\"])\n",
    "        logging.info(\"confusion matrix annotations: %s\", annotations)\n",
    "        metricsc.log_confusion_matrix(\n",
    "            annotations,\n",
    "            test_confusion_matrix[\"rows\"],\n",
    "        )\n",
    "\n",
    "        # log textual metrics info as well\n",
    "        for metric in metrics_list[0].keys():\n",
    "            if metric != \"confidenceMetrics\":\n",
    "                val_string = json.dumps(metrics_list[0][metric])\n",
    "                metrics.log_metric(metric, val_string)\n",
    "        # metrics.metadata[\"model_type\"] = \"AutoML Tabular classification\"\n",
    "\n",
    "    logging.getLogger().setLevel(logging.INFO)\n",
    "    aiplatform.init(project=project)\n",
    "    # extract the model resource name from the input Model Artifact\n",
    "    model_resource_path = model.uri.replace(\"aiplatform://v1/\", \"\")\n",
    "    logging.info(\"model path: %s\", model_resource_path)\n",
    "\n",
    "    client_options = {\"api_endpoint\": api_endpoint}\n",
    "    # Initialize client that will be used to create and send requests.\n",
    "    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)\n",
    "    eval_name, metrics_list, metrics_str_list = get_eval_info(\n",
    "        client, model_resource_path\n",
    "    )\n",
    "    logging.info(\"got evaluation name: %s\", eval_name)\n",
    "    logging.info(\"got metrics list: %s\", metrics_list)\n",
    "    log_metrics(metrics_list, metricsc)\n",
    "\n",
    "    thresholds_dict = json.loads(thresholds_dict_str)\n",
    "    deploy = classification_thresholds_check(metrics_list[0], thresholds_dict)\n",
    "    if deploy:\n",
    "        dep_decision = \"true\"\n",
    "    else:\n",
    "        dep_decision = \"false\"\n",
    "    logging.info(\"deployment decision is %s\", dep_decision)\n",
    "\n",
    "    return (dep_decision,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "459d9d60",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "automl-tab-chicago-taxi-tips-1624573137\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "DISPLAY_NAME = \"automl-tab-chicago-taxi-tips-{}\".format(str(int(time.time())))\n",
    "print(DISPLAY_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5e020d5",
   "metadata": {},
   "source": [
    "### Define the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "06111abc",
   "metadata": {},
   "outputs": [],
   "source": [
    "@kfp.dsl.pipeline(name=\"automl-tab-chicago-taxi-tips-train\", pipeline_root=PIPELINE_ROOT)\n",
    "def pipeline(\n",
    "    bq_source: str = \"bq://dougkelly-vertex-demos:chicago_taxi.chicago_taxi_tips_raw\",\n",
    "    display_name: str = DISPLAY_NAME,\n",
    "    project: str = PROJECT_ID,\n",
    "    gcp_region: str = REGION,\n",
    "    api_endpoint: str = \"us-central1-aiplatform.googleapis.com\",\n",
    "    thresholds_dict_str: str = '{\"auRoc\": 0.90}',\n",
    "):\n",
    "    dataset_create_op = gcc_aip.TabularDatasetCreateOp(\n",
    "        project=project, display_name=display_name, bq_source=bq_source\n",
    "    )\n",
    "\n",
    "    training_op = gcc_aip.AutoMLTabularTrainingJobRunOp(\n",
    "        project=project,\n",
    "        display_name=display_name,\n",
    "        optimization_prediction_type=\"classification\",\n",
    "        optimization_objective=\"maximize-au-roc\", # binary classification        \n",
    "        budget_milli_node_hours=750,\n",
    "        training_fraction_split=0.9,\n",
    "        validation_fraction_split=0.1,\n",
    "        column_transformations=[\n",
    "            {\"numeric\": {\"column_name\": \"trip_seconds\"}},            \n",
    "            {\"numeric\": {\"column_name\": \"trip_miles\"}},            \n",
    "            {\"categorical\": {\"column_name\": \"trip_month\"}},\n",
    "            {\"categorical\": {\"column_name\": \"trip_day\"}},\n",
    "            {\"categorical\": {\"column_name\": \"trip_day_of_week\"}},            \n",
    "            {\"categorical\": {\"column_name\": \"trip_hour\"}},            \n",
    "            {\"categorical\": {\"column_name\": \"payment_type\"}},\n",
    "            {\"numeric\": {\"column_name\": \"euclidean\"}},\n",
    "            {\"categorical\": {\"column_name\": \"tip_bin\"}},\n",
    "        ],\n",
    "        dataset=dataset_create_op.outputs[\"dataset\"],\n",
    "        target_column=\"tip_bin\",\n",
    "    )\n",
    "    \n",
    "    model_eval_task = classif_model_eval_metrics(\n",
    "        project,\n",
    "        gcp_region,\n",
    "        api_endpoint,\n",
    "        thresholds_dict_str,\n",
    "        training_op.outputs[\"model\"],\n",
    "    )\n",
    "\n",
    "    with dsl.Condition(\n",
    "        model_eval_task.outputs[\"dep_decision\"] == \"true\",\n",
    "        name=\"deploy_decision\",\n",
    "    ):\n",
    "\n",
    "        deploy_op = gcc_aip.ModelDeployOp(  # noqa: F841\n",
    "            model=training_op.outputs[\"model\"],\n",
    "            project=project,\n",
    "            machine_type=\"n1-standard-4\",\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "807aef93",
   "metadata": {},
   "source": [
    "### Compile and run the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "aaa6bb9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from kfp.v2 import compiler  # noqa: F811\n",
    "\n",
    "compiler.Compiler().compile(\n",
    "    pipeline_func=pipeline, package_path=\"automl-tab-chicago-taxi-tips-train_pipeline.json\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c4a344b",
   "metadata": {},
   "source": [
    "### Run the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "7b8ba21d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from kfp.v2.google.client import AIPlatformClient  # noqa: F811\n",
    "\n",
    "api_client = AIPlatformClient(project_id=PROJECT_ID, region=REGION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "1b7e85af",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "See the Pipeline job <a href=\"https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/automl-tab-chicago-taxi-tips-train-20210624221907?project=dougkelly-vertex-demos\" target=\"_blank\" >here</a>."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = api_client.create_run_from_job_spec(\n",
    "    \"automl-tab-chicago-taxi-tips-train_pipeline.json\",\n",
    "    pipeline_root=PIPELINE_ROOT,\n",
    "    parameter_values={\"project\": PROJECT_ID, \"display_name\": DISPLAY_NAME},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62c1108e",
   "metadata": {},
   "source": [
    "### Query your deployed model to retrieve online predictions and explanations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "ca2ee0a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.cloud import aiplatform\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "288537bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint = aiplatform.Endpoint(\n",
    "    endpoint_name=\"2677161280053182464\",\n",
    "    project=PROJECT_ID,\n",
    "    location=REGION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "6c15833e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Query complete after 0.00s: 100%|██████████| 1/1 [00:00<00:00, 286.20query/s]                          \n",
      "Downloading: 100%|██████████| 3096/3096 [00:01<00:00, 2959.44rows/s]\n"
     ]
    }
   ],
   "source": [
    "%%bigquery test_df\n",
    "\n",
    "SELECT\n",
    "  CAST(trip_month AS STRING) AS trip_month,\n",
    "  CAST(trip_day AS STRING) AS trip_day,\n",
    "  CAST(trip_day_of_week AS STRING) AS trip_day_of_week,\n",
    "  CAST(trip_hour AS STRING) AS trip_hour,\n",
    "  CAST(trip_seconds AS STRING) AS trip_seconds,\n",
    "  trip_miles,\n",
    "  payment_type,\n",
    "  euclidean\n",
    "FROM \n",
    "  dougkelly-vertex-demos.chicago_taxi.chicago_taxi_tips_raw\n",
    "WHERE \n",
    "  data_split = 'TEST'\n",
    "  AND tip_bin = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "f1d88288",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_instance = test_df.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "31d500e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'trip_month': '10',\n",
       " 'trip_day': '11',\n",
       " 'trip_day_of_week': '1',\n",
       " 'trip_hour': '16',\n",
       " 'trip_seconds': '360',\n",
       " 'trip_miles': 1.0,\n",
       " 'payment_type': 'Credit Card',\n",
       " 'euclidean': 0.0}"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_instance_dict = test_instance.to_dict()\n",
    "test_instance_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "425d6ab6",
   "metadata": {},
   "outputs": [],
   "source": [
    "explained_prediction = endpoint.explain([test_instance_dict])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "03015513",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD1CAYAAABA+A6aAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAARx0lEQVR4nO3db4xe5Xnn8e8v/hNMmrQFvIXYGDutCbgBQ3dsNlE3ZWkwpmziRttoTfNHS7ayvFkn4UUinEjhTaQIlF1pswnNyEJea5WoZiXaYsEUslRpSzagjtmwpoZARm4xs+4qDmmAsHjNkGtfzCR9eJg/Z5wZj337+5FGeu77vjjPNaPxz4fb55wnVYUk6fT3hoVuQJI0Nwx0SWqEgS5JjTDQJakRBrokNcJAl6RGLF6oNz7vvPNq9erVC/X2knRaevTRR39QVcsnW+sU6Ek2A18CFgF3VtVtfeu/CHwNWDVxzP9QVf9lumOuXr2a/fv3d3l7SdKEJM9MtTbjlkuSRcAdwPXAOuDGJOv6yv498ERVrQeuBv5jkqUn3LEkada67KFvBEaq6lBVHQf2Alv6agp4c5IAvwD8EBib004lSdPqEugrgGd7xqMTc72+AlwKHAEeBz5ZVT/pP1CSbUn2J9l/9OjRE2xZkjSZLnvomWSu/wEw1wGPAdcAvwr89yQPVdULr/mPqnYBuwAGBgZe9xCZV155hdHRUY4dO9ahrbadddZZrFy5kiVLlix0K5JOE10CfRS4sGe8kvEz8V43AbfV+JO+RpL8LXAJ8NezaWZ0dJQ3v/nNrF69mvHdmzNTVfHcc88xOjrKmjVrFrodSaeJLlsuw8DaJGsm/qFzK7Cvr+Yw8NsASX4FeDtwaLbNHDt2jHPPPfeMDnOAJJx77rn+n4qkWZnxDL2qxpLsAB5g/LLF3VV1MMn2ifVB4PPAniSPM75Fc0tV/eBEGjrTw/yn/DlImq1O16FX1RAw1Dc32PP6CLBpbluTJM3Ggt0p2sXqnffN6fH+7rYb5vR4MxkbG2Px4lP6R6zT2Fz/+TjTnex8mA8+y6XPSy+9xA033MD69et5xzvewV133cXw8DDvete7WL9+PRs3buTFF1/k2LFj3HTTTVx22WVceeWVfPOb3wRgz549fOADH+C9730vmzZt4qWXXuKjH/0oGzZs4Morr+See+4B4ODBg2zcuJErrriCyy+/nO9973sL+W1LaoCnj33uv/9+3vrWt3LffeNnP88//zxXXnkld911Fxs2bOCFF15g2bJlfOlLXwLg8ccf57vf/S6bNm3i6aefBuDhhx/mwIEDnHPOOXz2s5/lmmuuYffu3fzoRz9i48aNvOc972FwcJBPfvKTfPCDH+T48eO8+uqrC/Y9S2qDZ+h9LrvsMh588EFuueUWHnroIQ4fPswFF1zAhg0bAHjLW97C4sWL+da3vsWHP/xhAC655BIuuuiinwX6tddeyznnnAPAN77xDW677TauuOIKrr76ao4dO8bhw4d55zvfyRe+8AVuv/12nnnmGZYtW7Yw37CkZniG3ufiiy/m0UcfZWhoiM985jNs2rRp0itOpvtw7Te96U2vqbv77rt5+9vf/pqaSy+9lKuuuor77ruP6667jjvvvJNrrrlm7r4RSWccz9D7HDlyhLPPPpsPfehDfOpTn+KRRx7hyJEjDA8PA/Diiy8yNjbGu9/9br7+9a8D8PTTT3P48OHXhTbAddddx5e//OWf/QXwne98B4BDhw7xtre9jU984hO8733v48CBAyfpO5TUKs/Q+zz++ON8+tOf5g1veANLlizhq1/9KlXFxz/+cV5++WWWLVvGgw8+yMc+9jG2b9/OZZddxuLFi9mzZw9vfOMbX3e8z33uc9x8881cfvnlVBWrV6/m3nvv5a677uJrX/saS5Ys4fzzz+fWW29dgO9WUksy3dbBfBoYGKj+56E/+eSTXHrppQvSz6nIn4em42WLc+t0uWwxyaNVNTDZmlsuktQIA12SGmGgS1IjTrlAX6g9/VONPwdJs3VKBfpZZ53Fc889d8aH2U+fh37WWWctdCuSTiOn1GWLK1euZHR0FD+e7h8/sUiSujqlAn3JkiV+Qo8knaBTastFknTiDHRJakSnQE+yOclTSUaS7Jxk/dNJHpv4+pskryY5Z+7blSRNZcZAT7IIuAO4HlgH3JhkXW9NVX2xqq6oqiuAzwB/WVU/nId+JUlT6HKGvhEYqapDVXUc2Atsmab+RuCP5qI5SVJ3XQJ9BfBsz3h0Yu51kpwNbAbu/vlbkyTNRpdAf/2nO8BUd/68F/gfU223JNmWZH+S/V5rLklzq0ugjwIX9oxXAkemqN3KNNstVbWrqgaqamD58uXdu5QkzahLoA8Da5OsSbKU8dDe11+U5BeB3wLumdsWJUldzHinaFWNJdkBPAAsAnZX1cEk2yfWBydK3w98o6pemrduJUlT6nTrf1UNAUN9c4N94z3AnrlqTJI0O94pKkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEZ0CPcnmJE8lGUmyc4qaq5M8luRgkr+c2zYlSTOZ8TNFkywC7gCuBUaB4ST7quqJnppfAv4Q2FxVh5P8k3nqV5I0hS5n6BuBkao6VFXHgb3Alr6a3wf+uKoOA1TV9+e2TUnSTLoE+grg2Z7x6MRcr4uBX07yF0keTfKRuWpQktTNjFsuQCaZq0mO80+B3waWAQ8neaSqnn7NgZJtwDaAVatWzb5bSdKUupyhjwIX9oxXAkcmqbm/ql6qqh8AfwWs7z9QVe2qqoGqGli+fPmJ9ixJmkSXQB8G1iZZk2QpsBXY11dzD/DPkyxOcjZwFfDk3LYqSZrOjFsuVTWWZAfwALAI2F1VB5Nsn1gfrKonk9wPHAB+AtxZVX8zn41Lkl6ryx46VTUEDPXNDfaNvwh8ce5akyTNhneKSlIjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqRKdAT7I5yVNJRpLsnGT96iTPJ3ls4uvWuW9VkjSdGT9TNMki4A7gWmAUGE6yr6qe6Ct9qKr+5Tz0KEnqoMsZ+kZgpKoOVdVxYC+wZX7bkiTNVpdAXwE82zMenZjr984k/yvJnyX59ckOlGRbkv1J9h89evQE2pUkTaVLoGeSueob/0/goqpaD3wZ+NPJDlRVu6pqoKoGli9fPqtGJUnT6xLoo8CFPeOVwJHegqp6oap+PPF6CFiS5Lw561KSNKMugT4MrE2yJslSYCuwr7cgyflJMvF648Rxn5vrZiVJU5vxKpeqGkuyA3gAWATsrqqDSbZPrA8Cvwf8uyRjwMvA1qrq35aRJM2jGQMdfraNMtQ3N9jz+ivAV+a2NUnSbHinqCQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDWiU6An2ZzkqSQjSXZOU7chyatJfm/uWpQkdTFjoCdZBNwBXA+sA25Msm6KutsZ/+xRSdJJ1uUMfSMwUlWHquo4sBfYMkndx4G7ge/PYX+SpI66BPoK4Nme8ejE3M8kWQG8HxhEkrQgugR6JpmrvvF/Am6pqlenPVCyLcn+JPuPHj3asUVJUheLO9SMAhf2jFcCR/pqBoC9SQDOA34nyVhV/WlvUVXtAnYBDAwM9P+lIEn6OXQJ9GFgbZI1wP8GtgK/31tQVWt++jrJHuDe/jCXJM2vGQO9qsaS7GD86pVFwO6qOphk+8S6++aSdArocoZOVQ0BQ31zkwZ5Vf2bn78tSdJseaeoJDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGdAr0JJuTPJVkJMnOSda3JDmQ5LEk+5P85ty3KkmazoyfKZpkEXAHcC0wCgwn2VdVT/SU/Tmwr6oqyeXAfwMumY+GJUmT63KGvhEYqapDVXUc2Ats6S2oqh9XVU0M3wQUkqSTqkugrwCe7RmPTsy9RpL3J/kucB/w0blpT5LUVZdAzyRzrzsDr6o/qapLgN8FPj/pgZJtE3vs+48ePTqrRiVJ0+sS6KPAhT3jlcCRqYqr6q+AX01y3iRru6pqoKoGli9fPutmJUlT6xLow8DaJGuSLAW2Avt6C5L8WpJMvP4NYCnw3Fw3K0ma2oxXuVTVWJIdwAPAImB3VR1Msn1ifRD4V8BHkrwCvAz8655/JJUknQQzBjpAVQ0BQ31zgz2vbwdun9vWJEmz4Z2iktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNaLT89DPZKt33rfQLTTl7267YaFbkJrlGbokNaJToCfZnOSpJCNJdk6y/sEkBya+vp1k/dy3KkmazoyBnmQRcAdwPbAOuDHJur6yvwV+q6ouBz4P7JrrRiVJ0+tyhr4RGKmqQ1V1HNgLbOktqKpvV9U/TAwfAVbObZuSpJl0CfQVwLM949GJuan8W+DPfp6mJEmz1+Uql0wyV5MWJv+C8UD/zSnWtwHbAFatWtWxRUlSF13O0EeBC3vGK4Ej/UVJLgfuBLZU1XOTHaiqdlXVQFUNLF++/ET6lSRNoUugDwNrk6xJshTYCuzrLUiyCvhj4MNV9fTctylJmsmMWy5VNZZkB/AAsAjYXVUHk2yfWB8EbgXOBf4wCcBYVQ3MX9uSpH6d7hStqiFgqG9usOf1HwB/MLetSZJmwztFJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1olOgJ9mc5KkkI0l2TrJ+SZKHk/y/JJ+a+zYlSTOZ8TNFkywC7gCuBUaB4ST7quqJnrIfAp8Afnc+mpQkzazLGfpGYKSqDlXVcWAvsKW3oKq+X1XDwCvz0KMkqYMugb4CeLZnPDoxJ0k6hXQJ9EwyVyfyZkm2JdmfZP/Ro0dP5BCSpCl0CfRR4MKe8UrgyIm8WVXtqqqBqhpYvnz5iRxCkjSFLoE+DKxNsibJUmArsG9+25IkzdaMV7lU1ViSHcADwCJgd1UdTLJ9Yn0wyfnAfuAtwE+S3Aysq6oX5q91SVKvGQMdoKqGgKG+ucGe1/+H8a0YSdIC8U5RSWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmN6BToSTYneSrJSJKdk6wnyX+eWD+Q5DfmvlVJ0nRmDPQki4A7gOuBdcCNSdb1lV0PrJ342gZ8dY77lCTNoMsZ+kZgpKoOVdVxYC+wpa9mC/Bfa9wjwC8luWCOe5UkTWNxh5oVwLM941Hgqg41K4C/7y1Kso3xM3iAHyd5albdajrnAT9Y6CZmktsXugMtAH8359ZFUy10CfRMMlcnUENV7QJ2dXhPzVKS/VU1sNB9SP383Tx5umy5jAIX9oxXAkdOoEaSNI+6BPowsDbJmiRLga3Avr6afcBHJq52+WfA81X19/0HkiTNnxm3XKpqLMkO4AFgEbC7qg4m2T6xPggMAb8DjAD/F7hp/lrWFNzK0qnK382TJFWv2+qWJJ2GvFNUkhphoEtSIwx0SWpEl+vQdQpKcgnjd+iuYPya/yPAvqp6ckEbk7RgPEM/DSW5hfFHMAT4a8YvLQ3wR5M9PE06FSTx6rd55lUup6EkTwO/XlWv9M0vBQ5W1dqF6UyaWpLDVbVqoftomVsup6efAG8Fnumbv2BiTVoQSQ5MtQT8ysns5UxkoJ+ebgb+PMn3+MeHoq0Cfg3YsVBNSYyH9nXAP/TNB/j2yW/nzGKgn4aq6v4kFzP+aOMVjP9hGQWGq+rVBW1OZ7p7gV+oqsf6F5L8xUnv5gzjHrokNcKrXCSpEQa6JDXCQJekRhjoktQIA12SGvH/ARAp2qqRLlJ5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.DataFrame.from_dict(explained_prediction.predictions[0]).plot(kind='bar');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "970886b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAboAAAD4CAYAAACaECNWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeZUlEQVR4nO3deZhcdZ3v8feHEAwQNCO0DBqgMaJIWCJpAkR0ANeLC4PGCcJFAZWrOAp3AI0yOqjDNeh9dAZRMfAIboCKRhwYFkFCWEKSDiTpEJEtzBD0ahMFCbIZPvePOi1Fp7q6Ot3VVX3yeT1PP3Xq9/ud3/nWyfLps1SVbBMREVFWW7S6gIiIiGZK0EVERKkl6CIiotQSdBERUWoJuoiIKLUtW11APN8OO+zgzs7OVpcRETGmLFu27GHbHbX6EnRtprOzk+7u7laXERExpkj6r4H6cuoyIiJKLUEXERGllqCLiIhSyzW6iIgA4JlnnmHt2rU8+eSTrS5lQBMmTGDy5MmMHz++4XUSdBERAcDatWvZbrvt6OzsRFKry9mIbdatW8fatWvZbbfdGl4vQVcynXOuHLDvgblvG8VKImKsefLJJ9s25AAksf3229Pb2zuk9XKNLiIi/qpdQ67PptSXoIuIiFLLqcuIiKip3qWQTdHI5ZOrr76ak08+mQ0bNvDBD36QOXPmDHu7OaKLiIi2sGHDBj760Y9y1VVXsXr1ai655BJWr1497HlbGnSSJkk6qU7/raNZz0iR1ClpVavriIgYS5YsWcIrXvEKXv7yl7PVVltx1FFHcfnllw973lYf0U0CNgo6SeMAbM8c7YIiIqI1HnroIXbeeee/Pp88eTIPPfTQsOdtddDNBaZIWi5pqaQbJF0M9ABIWl88HiJpoaT5klZLOk9SzdoljZN0kaRVknok/e+ifYqkqyUtk3STpD2K9h2LeVcUPzOL9n8q5lgl6ZSirVPSrySdL+lOSddK2rrom16svwj4aFU9UyUtKV7jSkm716j5REndkrqHettsRERZ2N6obSTuAm110M0B7rM9DTgdmAGcYXvPGmNnAKcCewNTgHcNMOc04GW297K9N3Bh0T4P+Jjt6cBpwDeK9nOAG23vC+wH3ClpOnA8cABwIPAhSa8pxu8OfN32VOAR4N1F+4XAx20f1K+eDwP/XrzGLmBt/4Jtz7PdZburo6Pmt0xERJTe5MmTefDBB//6fO3atbz0pS8d9rytDrr+ltheU6fvftsbgEuAgwcYdz/wcklfk/RW4E+SJgIzgR9LWg58C9ipGH8Y8E0A2xtsP1rMPd/247bXAz8FXleMX2N7ebG8DOiU9CJgku0bi/bvVdWzCPi0pE8Cu9p+orFdERGxedl///255557WLNmDU8//TSXXnop73znO4c9b7u9veDxOn39j2k3PsYFbP9R0r7AW6icQvwH4BTgkeKoqhH1jpWfqlreAGxdjB+onoslLQbeBlwj6YO2f9lgHRERLTPan6a05ZZbcu655/KWt7yFDRs2cMIJJzB16tRhz9vqI7rHgO0aHDtD0m7FtbnZwM21BknaAdjC9k+AzwD72f4TsEbSe4oxKsIQ4HrgI0X7OEkvBBYCfy9pG0nbAkcCNw1UmO1HgEcl9R1lHlNVz8uB+22fA/wc2KfB1xsRsdk5/PDDufvuu7nvvvs444wzRmTOlgad7XXALcWt+F8eZPgiKjevrALWAPMHGPcyYEFxivIi4FNF+zHAByStAO4EjijaTwYOldRD5VTkVNu3F+suARYDF9i+Y5D6jge+XtyMUn16cjawqqhnD+C7g8wTEREjqOWnLm0fXadvYtXTP9ue3cB8K6jcVNK/fQ3w1hrtv+O50Ktu/wrwlX5tDwB7VT3/v1XLy4B9q4afWbR/EfjiYHVHRERztDzoYmTlGwoiYjhst/UHO9d6C8JgxkTQ2V4ALOjfXtzk8YJ+zcfa7hmFsiIiSmXChAmsW7eO7bffvi3Dru/76CZMmDCk9cZE0A3E9gGtriEioiwmT57M2rVrh/x9b6Op7xvGh2JMB11ERIyc8ePHD+mbu8eKVr+9ICIioqkSdBERUWoJuoiIKLUEXURElFqCLiIiSi1BFxERpZagi4iIUkvQRUREqSXoIiKi1PLJKJuRzjlXtrqEiIgBNetD6XNEFxERpZagi4iIUkvQRUREqW3WQSfpOEkvHWTMKZK2Ga2aIiJiZG3WQQccB9QNOuAUIEEXETFGNTXoJHVKukvSdyStlHSZpG0kfVbSUkmrJM1TxRRJt1etu7ukZcXyA5L+j6RFkrol7SfpGkn3Sfpw1TqnF/OulPS5qhp+Jel8SXdKulbS1pJmAV3ADyQtl7R1jfo/TiUIb5B0g6QPSPpqVf+HJH1loNdZjJku6UZJy4qad2rW/o6IiI2NxhHdq4B5tvcB/gScBJxre3/bewFbA2+3fR/wqKRpxXrHAxdVzfOg7YOAm4r2WcCBwOcBJL0Z2B2YAUwDpkt6fbHu7sDXbU8FHgHebfsyoBs4xvY020/0L9z2OcBvgENtHwpcCrxT0viqGi8c6HUW474GzLI9Hfg2cFb/7Ug6sQjw7nb+Zt+IiLFoNILuQdu3FMvfBw4GDpW0WFIPcBgwtei/ADhe0jhgNnBx1Tw/Lx57gMW2H7PdCzwpaRLw5uLnDuB2YA8qAQewxvbyYnkZ0LkpL8T248AvgbdL2gMYb7unzut8FbAX8AtJy4F/Bjb6Dnjb82x32e7q6OjYlNIiImIAo/GGcdd4/g2gy/aDks4EJhR9PwH+hUqYLLO9rmq9p4rHZ6uW+55vCQj4ou1vVW9MUme/8RuoHEVuqguATwN38dzRHNR+nQLuLI5EIyKiBUbjiG4XSX3/0b8XuLlYfljSRCqnIAGw/SRwDfBNnh8ijbgGOKGYE0kvk/SSQdZ5DNhuKGNsLwZ2Bo4GLqkaV+t1/hro6GuXNF7SVCIiYtSMRtD9Cni/pJXAi6mE2PlUTkH+DFjab/wPqBwNXTuUjdi+lsqpzkXFKdHLGDzELgLOG+hmlMI84CpJN1S1/Qi4xfYfq9o2ep22n6YS5GdLWgEsB2YO5XVFRMTwyO5/xm0EJ6+cNryiuOmk0XVOA15k+zNNK2yYJF0BfNX29cXzTob4OgfS1dXl7u7u4U5TUz7rMiLa2XA+61LSMttdtfra6kOdJc0HplC5QaXtFDe9LAFW9IVcRES0t6Ye0Y0lRcju1q/5k7avGc06mnlEFxFRVmPmiK6VbB/Z6hoiImLkbe4fARYRESWXoIuIiFJL0EVERKkl6CIiotQSdBERUWoJuoiIKLUEXURElFqCLiIiSi1BFxERpZagi4iIUkvQRUREqSXoIiKi1BJ0m5F8H11EbI4SdBERUWoJuoiIKLUEXURElNqYDDpJkySdVKf/1iZuu0vSOcXycZLObda2IiJi+MZk0AGTgI2CTtI4ANszm7Vh2922P96s+SMiYmSN1aCbC0yRtFzSUkk3SLoY6AGQtL54PETSQknzJa2WdJ6kAV+zpPWSzpa0TNJ1kmZIWiDpfknvrJrzihrrdkj6SVHPUkmvLdr/rqhzuaQ7JG1XY90TJXVL6u7t7R2ZPRQREcDYDbo5wH22pwGnAzOAM2zvWWPsDOBUYG9gCvCuOvNuCyywPR14DPhX4E3AkcDnB6np34Gv2t4feDdwQdF+GvDRotbXAU/0X9H2PNtdtrs6OjoG2UxERAzFlq0uYIQssb2mTt/9AJIuAQ4GLhtg7NPA1cVyD/CU7Wck9QCdg9TwRmBPSX3PX1gcvd0CfEXSD4Cf2l7byAuKiIiRUZage7xOnwd5Xu0Z2339zwJPAdh+VtJg+2oL4CDb/Y/Y5kq6EjgcuE3SG23fNchcERExQsbqqcvHgI2udQ1ghqTdimtzs4Gbm1TTtcA/9j2RNK14nGK7x/bZQDewR5O2HxERNYzJoLO9DrhF0irgy4MMX0Tl5pVVwBpgfpPK+jjQJWmlpNXAh4v2UyStkrSCyvW5q5q0/YiIqGHMnrq0fXSdvolVT/9se3aDc06sWj6zVp/tBcCCYvki4KJi+WEqR4z95/xYI9uOiIjmGJNHdLFpHpj7tlaXEBEx6sbsEV0jqo++qklaDLygX/OxtntGoayIiBhFpQ66gdg+oNU1RETE6Mipy4iIKLUEXURElFqCLiIiSi1BFxERpZagi4iIUkvQRUREqSXoIiKi1BJ0ERFRagm6iIgotQRdRESUWoIuIiJKLUEXERGllqCLiIhSS9BFRESpJegiIqLUShl0kiZJOqlO/60jsI3jJJ073HkiIqK5Shl0wCRgo6CTNA7A9szRLigiIlqjrEE3F5giabmkpZJukHQx0AMgaX3xeIikhZLmS1ot6TxJA+4TScdLulvSjcBrq9rfIWmxpDskXSdpR0lbSLpHUkcxZgtJ90raoca8J0rqltTd29s7wrsiImLzVtagmwPcZ3sacDowAzjD9p41xs4ATgX2BqYA76o1oaSdgM9RCbg3AdVz3QwcaPs1wKXAJ2w/C3wfOKYY80Zghe2H+89te57tLttdHR0dQ32tERFRR1mDrr8lttfU6bvf9gbgEuDgAcYdACyw3Wv7aeCHVX2TgWsk9VAJ1qlF+7eB9xXLJwAXDudFRETE0G0uQfd4nT4P8ryRvq8B59reG/hfwAQA2w8Cv5N0GJWgvKqxciMiYqSUNegeA7ZrcOwMSbsV1+ZmUzkNWcti4BBJ20saD7ynqu9FwEPF8vv7rXcBlVOYPyqOGiMiYhSVMuhsrwNukbQK+PIgwxdRuXllFbAGmD/AnL8FzizGXwfcXtV9JvBjSTcB/a/B/RyYSE5bRkS0xJatLqBZbB9dp29i1dM/257d4JwXUiOwbF8OXD7AavtSuQnlrka2ERERI6u0QdcOJM0BPsJzd15GRMQo26yDzvYCYEH/dkmLgRf0az7Wds8Q559L5bRoRES0yGYddAOxfUCra4iIiJFRyptRIiIi+iToIiKi1BJ0ERFRagm6iIgotQRdRESUWoIuIiJKLUEXERGllqCLiIhSS9BFRESpJegiIqLUEnQREVFqCbrNSOecK+mcc2Wry4iIGFUJuoiIKLUEXURElFqCLiIiSm2zDjpJ64vHl0q6bIAxCyR1jW5lERExUvLFq4Dt3wCzWl1HRESMvDF7RCfpf0paImm5pG9JGtd3hFb0z5J0UbG8o6T5klYUPzP7zdUpaVWxvLWkSyWtlPRDYOuqcW+WtEjS7ZJ+LGli0f5ZSUslrZI0T5KK9gWSzi7qvFvS65q/ZyIiotqYDDpJrwZmA6+1PQ3YABxTZ5VzgBtt7wvsB9xZZ+xHgD/b3gc4C5hebHMH4J+BN9reD+gG/qlY51zb+9vei0owvr1qvi1tzwBOAf5lgNdzoqRuSd29vb11SouIiKEaq6cu30AlgJYWB09bA7+vM/4w4H0AtjcAj9YZ+3oqwYjtlZJWFu0HAnsCtxTb3ApYVPQdKukTwDbAi6kE6X8UfT8tHpcBnbU2aHseMA+gq6vLdWqLiIghGqtBJ+A7tj/1vEbp1KqnE4Yxf62wEfAL2+/tt80JwDeALtsPSjqz37afKh43MHb3d0TEmDUmT10C1wOzJL0EQNKLJe0K/E7SqyVtARzZb/xHirHjJL2wztwLKU6DStoL2Kdovw14raRXFH3bSHolz4Xaw8U1u9zUEhHRRsZk0NleTeV62bXFqcVfADsBc4ArgF8Cv61a5WQqpxd7qJxCnFpn+m8CE4t5PwEsKbbZCxwHXFL03QbsYfsR4HygB/gZsHREXmRERIwI2bkk1E66urrc3d3dlLn7Pufygblva8r8ERGtImmZ7Zrvec41o81IAi4iNkdj8tRlREREoxJ0ERFRagm6iIgotQRdRESUWoIuIiJKLUEXERGllqCLiIhSS9BFRESpJegiIqLUEnQREVFqCbqIiCi1BF1ERJRagi4iIkot315Qcn1fzQP59oKI2DzliC4iIkotQRcREaWWoIuIiFJL0EVERKnVDTpJkySdVKf/1uEWIOk4SecOd54GttMhabGkOyS9rtnbK7Z5kaRZo7GtiIiobbAjuknARkEnaRyA7ZlNqKlZ3gDcZfs1tm9qdTERETE6Bgu6ucAUScslLZV0g6SLgR4ASeuLx0MkLZQ0X9JqSedJGnBuScdLulvSjcBrq9rfUXXUdZ2kHSVtIekeSR3FmC0k3StphwHm3lXS9ZJWFo+7SJoGfAk4vHgtW9dY7x8kfaVYPlnS/cXyFEk3F8vTJd0oaZmkayTtVDXm6qL9Jkl71Jj/C8UR3kb7RdKJkroldff29g602yIiYhMMFnRzgPtsTwNOB2YAZ9jes8bYGcCpwN7AFOBdtSYswuFzVALuTUD1XDcDB9p+DXAp8AnbzwLfB44pxrwRWGH74QFqPhf4ru19gB8A59heDnwW+KHtabafqLHeQqDvlObrgHWSXgYcDNwkaTzwNWCW7enAt4GzivHzgI8V7acB3+j3mr8EvAQ4vng9z2N7nu0u210dHR0DvKyIiNgUQ33D+BLba+r09R0FXUIlIC6rMe4AYIHt3mLsD4FXFn2TgR8WYbgV0LetbwOXA/8GnABcWKfGg3guZL9H5UhuULb/n6SJkrYDdgYuBl5PJfR+CrwK2Av4hSSAccBvJU0EZgI/LtoBXlA19WeAxbZPbKSOiIgYWUMNusfr9HmQ5430fQ34iu2fSzoEOBPA9oOSfifpMCpBecwA6w9lW7UsAo4Hfg3cRCVUD6JypLoLcKftg6pXkPRC4JHiqLeWpcB0SS+2/Ych1BIRESNgsFOXjwHbNTjXDEm7FdegZlM5DVnLYuAQSdsXpwPfU9X3IuChYvn9/da7gMopzB/Z3lCnjluBo4rlY+rUUctCKqceFwJ3AIcCT9l+lEr4dUg6CEDSeElTbf8JWCPpPUW7JO1bNefVVK51XlkcLUZExCiqG3S21wG3SFoFfHmQuRZR+Q99FZVTjvMHmPO3VI7UFgHXAbdXdZ9J5RTgTUD/a3A/ByZS/7QlwMeB4yWtBI4FTh5kfLWbqJy2XFiE6YMUQWn7aWAWcLakFcByKqcsoRKoHyja7wSOqJ7U9o+B84Gf17oRJiIimkf2UM7sDTBJ5TTjabbfPuzJBt5GF/BV26PyHrhW6erqcnd394jNlw91jojNgaRltrtq9Y2Jby+QNAf4CEO7Nhck3CIiRiTobC8AFvRvl7SY59+BCHCs7Z4hzj+XymnR6rnP4PnX9wB+bPssBjFSdUVERPtr6hGd7QOaOPdZPPc+tqGu27S6IiKiveRDnSMiotQSdBERUWoJuoiIKLUEXURElFqCLiIiSi1BFxERpZagi4iIUkvQRUREqSXoIiKi1BJ0ERFRamPiQ51jZFR/k0GffOhzRJRdjugiIqLUEnQREVFqCbqIiCi1BF1ERJRaKYJO0iRJJ9Xpv3UT512/6VVFREQ7KEXQAZOAjYJO0jgA2zNHu6CB9NUUERGjoyxBNxeYImm5pKWSbpB0MdADzx2ZSTpE0kJJ8yWtlnSepLr7QNJZklZIuk3SjkXbrpKul7SyeNylaL9I0qyqdau3+7ya+m3jREndkrp7e3tHaJdERASUJ+jmAPfZngacDswAzrC9Z42xM4BTgb2BKcC76sy7LXCb7X2BhcCHivZzge/a3gf4AXBOAzUOWJPteba7bHd1dHQ0MFVERDSqLEHX3xLba+r03W97A3AJcHCdeZ4GriiWlwGdxfJBwMXF8vcGmaORmiIioknKGnSP1+nzIM+rPWO7r38DA3+STN+Yv1DsU0kCtmqwpoiIaJKyBN1jwHYNjp0habfi2txs4OZN2N6twFHF8jFVczwATC+WjwDGb8LcERExgkrxWZe210m6RdIq4Angd3WGL6Jy88reVK67zd+ETX4c+Lak04Fe4Pii/XzgcklLgOvJUVxERMuVIugAbB9dp29i1dM/257d4JwTq5YvAy4rlh8ADqsx/nfAgVVNnyraFwALGtlmRESMrNIEXQwu31QQEZujzSroBjqykrQYeEG/5mNtb/Set4iIGFs2q6AbiO0DWl1DREQ0R1nuuoyIiKgpQRcREaWWoIuIiFJL0EVERKkl6CIiotQSdBERUWoJuoiIKLUEXURElFqCLiIiSi1BFxERpZagi4iIUkvQRUREqSXoIiKi1BJ0ERFRagm6iIgotbYOOkmTJJ1Up//W0axnMJI+XbXcKWlVK+uJiIg2DzpgErBR0EkaB2B75mgXNIhPDz4kIiJGU7sH3VxgiqTlkpZKukHSxUAPgKT1xeMhkhZKmi9ptaTzJA342iStl3S2pGWSrpM0Q9ICSfdLemcxZoKkCyX1SLpD0qFF+3GSfirpakn3SPpS0T4X2Lqo9QfFpsZJOl/SnZKulbT1APWcKKlbUndvb+9I7buIiKD9g24OcJ/tacDpwAzgDNt71hg7AzgV2BuYAryrzrzbAgtsTwceA/4VeBNwJPD5YsxHAWzvDbwX+I6kCUXfNGB2sa3Zkna2PQd4wvY028cU43YHvm57KvAI8O5axdieZ7vLdldHR0edsiMiYqjaPej6W2J7TZ2++21vAC4BDq4zz9PA1cVyD3Cj7WeK5c6i/WDgewC27wL+C3hl0Xe97UdtPwmsBnYdYDtrbC8vlpdVzR0REaNkrAXd43X6PMjzas/Y7ut/FngKwPazwJZFu+qs/1TV8oaqdTZ1XERENEm7B91jwHYNjp0habfi2txs4OZhbnshcAyApFcCuwC/HmSdZySNH+Z2IyJiBLV10NleB9xS3Kb/5UGGL6Jy88oqYA0wf5ib/waVm0l6gB8Cx9l+apB15gErq25GiYiIFtNzZ/DGLkmHAKfZfnuLSxm2rq4ud3d3t7qMiIgxRdIy2121+tr6iC4iImK4SnFzhO0FwIL+7ZIWAy/o13ys7Z5RKCsiItpAKYJuILYPaHUNERHRWjl1GRERpZagi4iIUkvQRUREqSXoIiKi1ErxProykdRL5XM1m2EH4OEmzT0c7VhXampcO9aVmhrXjnVtSk272q75qfgJus2IpO6B3lDZSu1YV2pqXDvWlZoa1451jXRNOXUZERGllqCLiIhSS9BtXua1uoABtGNdqalx7VhXampcO9Y1ojXlGl1ERJRajugiIqLUEnQREVFqCboSkvRWSb+WdK+kOTX6Jemcon+lpP3aoKY9JC2S9JSk05pdzxDqOqbYRysl3Spp3zao6YiinuWSuiUd3OqaqsbtL2mDpFnNrqmRuiQdIunRYl8tl/TZVtdUVddySXdKurHVNUk6vWofrSr+DF/cBnW9SNJ/SFpR7KvjN2lDtvNToh9gHHAf8HJgK2AFsGe/MYcDVwECDgQWt0FNLwH2B86i8iW67bKvZgJ/Uyz/jzbZVxN57vr6PsBdra6patwvgf8EZrXJn98hwBWj8fdpCDVNAlYDuxTPX9LqmvqNfwfwyzbZV58Gzi6WO4A/AFsNdVs5oiufGcC9tu+3/TRwKXBEvzFHAN91xW3AJEk7tbIm27+3vRR4pol1bEpdt9r+Y/H0NmByG9S03sW/fGBboNl3lDXydwrgY8BPgN83uZ6h1jWaGqnpaOCntv8bKn/326Cmau8FLmlyTY3WZWA7SaLyC94fgL8MdUMJuvJ5GfBg1fO1RdtQx4x2Ta0w1Lo+QOVIuJkaqknSkZLuAq4ETmh1TZJeBhwJnNfkWoZUV+Gg4tTXVZKmtkFNrwT+RtICScskva8NagJA0jbAW6n8wtJsjdR1LvBq4DdAD3Cy7WeHuqFSf/HqZko12vr/xt/ImJE02ttrVMN1STqUStA1+3pYQzXZng/Ml/R64AvAG1tc078Bn7S9ofLL96hopK7bqXwG4npJhwM/A3ZvcU1bAtOBNwBbA4sk3Wb77hbW1OcdwC22/9CkWqo1UtdbgOXAYcAU4BeSbrL9p6FsKEd05bMW2Lnq+WQqvw0Ndcxo19QKDdUlaR/gAuAI2+vaoaY+thcCUyTt0OKauoBLJT0AzAK+Ienvm1hTQ3XZ/pPt9cXyfwLj22BfrQWutv247YeBhUAzb3Iayt+poxid05bQWF3HUznNa9v3AmuAPYa8pWZfcMzP6P5Q+W3xfmA3nrvAO7XfmLfx/JtRlrS6pqqxZzJ6N6M0sq92Ae4FZrZRTa/guZtR9gMe6nve6j+/YvxFjM7NKI3sq7+t2lczgP9u9b6iciru+mLsNsAqYK9W//kBL6JyDWzbZv/ZDWFffRM4s1jesfi7vsNQt5VTlyVj+y+S/hG4hspdTd+2faekDxf951G5K+5wKv+B/5nKb00trUnS3wLdwAuBZyWdQuUOrCGdohjpuoDPAttTOUIB+Iub+EnvDdb0buB9kp4BngBmu/ifoIU1jboG65oFfETSX6jsq6Nava9s/0rS1cBK4FngAturWllTMfRI4Frbjzerlk2o6wvARZJ6qPxi/klXjoKHJB8BFhERpZZrdBERUWoJuoiIKLUEXURElFqCLiIiSi1BFxERpZagi4iIUkvQRUREqf1/MQR6wk0q+HwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.DataFrame.from_dict(explained_prediction.explanations[0].attributions[0].feature_attributions, orient='index').plot(kind='barh');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8c01dd9",
   "metadata": {},
   "source": [
    "## Congratulations! Lab wrap-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0024688d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a93b6d2d",
   "metadata": {},
   "source": [
    "## License"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de1131c1",
   "metadata": {},
   "source": [
    "<font size=-1>Licensed under the Apache License, Version 2.0 (the \\\"License\\\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at [https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and limitations under the License.</font>"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "conda-root-py",
   "name": "tf2-gpu.2-6.m86",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m86"
  },
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
